Papers with word embedding methods
Domain-Specific Word Embeddings with Structure Prediction (2023.tacl-1)
Copied to clipboard
| Challenge: | Current word embedding methods do not provide a way to use or predict information on structure between sub-corpora, time or domain. |
| Approach: | They propose a word embedding method that provides general word representations for the whole corpus, domain-specific representations and embeddable alignment simultaneously. |
| Outcome: | The proposed method provides better performance than baselines on a dataset of science and philosophy articles. |
Embedding Words as Distributions with a Bayesian Skip-gram Model (C18-1)
Copied to clipboard
| Challenge: | Rather than assuming that word embeddings are fixed across the entire text collection, we generate them from word-specific prior densities for each word. |
| Approach: | They propose a method for embedding words as probability densities in a low-dimensional space from a word-specific prior density for each occurrence of a given word. |
| Outcome: | The proposed method can encode word as a distribution on a range of benchmarks and is comparable to Gaussian embeddings. |
Factors Influencing the Surprising Instability of Word Embeddings (N18-1)
Copied to clipboard
| Challenge: | Word embeddings are low-dimensional, dense vector representations that capture semantic properties of words. |
| Approach: | They examine the stability of word embeddings by examining their properties and analyzing their effects on downstream tasks. |
| Outcome: | The results show that even high frequency words exhibit substantial instability, which can have implications for downstream tasks. |
ANALOGICAL - A Novel Benchmark for Long Text Analogy Evaluation in Large Language Models (2023.findings-acl)
Copied to clipboard
Thilini Wijesiriwardene, Ruwan Wickramarachchi, Bimal Gajera, Shreeyash Gowaikar, Chandan Gupta, Aman Chadha, Aishwarya Naresh Reganti, Amit Sheth, Amitava Das
| Challenge: | Modern large language models are evaluated on extrinsic measures based on benchmarks such as GLUE and SuperGLUE. |
| Approach: | They propose a benchmark to intrinsically evaluate large language models across a taxonomy of analogies of long text with six levels of complexity. |
| Outcome: | The proposed benchmark evaluates LLMs across a taxonomy of analogies of long text with six levels of complexity. |
Learning Numeral Embedding (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing word embedding methods do not learn numeral embedds well because numerals are limited in number and their appearances in training corpora are highly scarce. |
| Approach: | They propose two numeral embedding methods that can handle the out-of-vocabulary problem for numerals. |
| Outcome: | The proposed methods can handle the out-of-vocabulary problem for numerals. |
Incorporating Syntactic and Semantic Information in Word Embeddings using Graph Convolutional Networks (P19-1)
Copied to clipboard
Shikhar Vashishth, Manik Bhandari, Prateek Yadav, Piyush Rai, Chiranjib Bhattacharyya, Partha Talukdar
| Challenge: | Existing word embedding methods utilize sequential context of a word to learn its embeddment, but such methods result in an explosion of the vocabulary size. |
| Approach: | They propose a flexible Graph Convolution based method for learning word embeddings that utilizes the dependency context of a word without increasing the vocabulary size. |
| Outcome: | The proposed model outperforms existing methods on intrinsic and extrinsic tasks and provides an advantage when used with ELMo. |
Invernet: An Inversion Attack Framework to Infer Fine-Tuning Datasets through Word Embeddings (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing word embeddings are data intensive and require large-scale training corpus, sufficient training iterations, and high computational capacity. |
| Approach: | They propose a framework that infers context distributions from a downstream dataset and then uses them to fine-tune the embedding. |
| Outcome: | The proposed framework materializes privacy concern by inferring context distribution in the downstream dataset, which can lead to key information breach. |
PWESuite: Phonetic Word Embeddings and Tasks They Facilitate (2024.lrec-main)
Copied to clipboard
Vilém Zouhar, Kalvin Chang, Chenxuan Cui, Nate B. Carlson, Nathaniel Romney Robinson, Mrinmaya Sachan, David R. Mortensen
| Challenge: | Existing word embedding methods overlook phonetic information that is crucial for many tasks. |
| Approach: | They propose three methods that use articulatory features to build phonetically informed word embeddings. |
| Outcome: | The proposed methods improve word retrieval and correlation with sound similarity and on rhyme and cognate detection tasks. |